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Why GAO Did This Study 

As a condition of receiving federal 
funding for elementary and 
secondary education programs, 
states each year provide vast 
amounts of data to Education. 
While the need for information that 
informs evaluation is important 
(particularly with the No Child Left 
Behind Act), Education’s data 
gathering has heretofore presented 
some problems. It has been 
burdensome to states because 
there are multiple and redundant 
requests administered by a number 
of offices. In addition, the resulting 
data supplied by states has not 
been accurate, timely, or conducive 
to assessing program performance. 
To improve the information by 
which it evaluates such programs 
and also to ease states’ reporting 
burden. Education in 2002 initiated 
an ambitious, multiyear plan to 
consolidate elementary and 
secondary data collections into a 
single, departmentwide system 
focused on performance. Given its 
importance, we prepared a study, 
under the authority of the 
Comptroller General, to provide 
Congress with information on its 
progress. 



What GAO Recommends 



GAO recommends that Education 
(1) develop a strategy to help states 
provide quality data, (2) develop a 
process within the department to 
resolve critical, outstanding issues, 
and (3) develop a clear plan for 
completing final aspects of PBDMI, 
including specific time frames and 
indicators of progress toward the 
initiative’s goals. Education agreed 
with our recommendations. 
www.gao.gov/cgi-bin/getrpt7GAO-06-6. 

To view the full product, including the scope 
and methodology, click on the link above. 

For more information, contact David Beilis at 
(415) 904-2272 or bellisd@gao.gov. 



What GAO Found 

Through its Performance-Based Data Management Initiative (PBDMI), 
Education has consolidated and defined much of the data it anticipates 
collecting under a unified system. Education reports that many data 
definitions have been agreed-to and data redundancies eliminated. PBDMI 
officials also said that to date, however, it has not been able to resolve all 
remaining differences among the program offices that manage many of the 
d iff erent data collections. 

PBDMI officials have conducted extensive outreach to the states to advance 
the initiative. The outreach to states involved regional conferences, two 
rounds of site visits, and according to officials, $100,000 in grants to most 
states to help offset their costs. State data providers responding to our 
survey expressed general satisfaction with the department’s outreach, but 
some were not optimistic that the initiative would ease their reporting 
burden or enhance their own analytic capacity. The states were not able to 
produce enough data during test submissions in 2003 and 2004 to enable 
data quality verification or phasing out the department’s multiple data 
collections. With regard to the lack of sufficient data from many states. 
Education officials said some lack the technical capacity needed to produce 
new performance data requirements. State data providers reported having 
competing demands for their time and resources, given other federal 
initiatives. 



Education officials have decided to proceed with the undertaking and have 
developed a draft interim strategy for moving forward. But they currently 
have no formal plan for how they would overcome obstacles such as the lack 
of state data and other technical and training delays to the initiative. 



Reporting to Education: A Sampie of Data Coiiections Seeking information on Eiementary 
and Secondary Programs in One State in 2004 




By Education’s own 
account, around 200 
data collections are 
administered to states 
about eiementary and 
secondary programs 



March 



May 



Aug. 



Oct. 



. Survey of private, not-tor profit schools 

• 2 common core ot data surveys: schools and state nonfiscal survey 

• 3 program assessments: Title I, vocational, and special education 
. Private schools participation in Title I 

• Follow-up report on career and technical education completers 

. Instruction tor limited English and immigrant students 
. School and statt admininistrative report 
. Discipline, crime, and violence report 
. Charter school evaluation report 

. Instructional personnel report 
. Student enrollment data collection (fall count) 

. Sate and drug free schools report 
. Count ot home-schooled children 



Source: Virginia State Department of Education. 
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October 28, 2005 

The Honorable Michael B. Enzi 
Chairman 

The Honorable Edward M. Kennedy 
Ranking Minority Member 
Committee on Health, Education, 

Labor, and Pensions 
United States Senate 

The Honorable John A. Boehner 
Chairman 

The Honorable George Miller 

Ranking Minority Member 

Committee on Education and the Workforce 

House of Representatives 

Each year, state education agencies provide vast amounts of information 
to the U.S. Department of Education (Education) in order to fulfill 
reporting requirements for federal programs supporting elementary and 
secondary education. While this information is important for managing 
programs, it has been accompanied by some problems. Reporting has been 
burdensome for the state data providers because the department makes its 
data requests through multiple, ongoing, and uncoordinated data 
collections. By Education’s own accoimt, there are currently 200 active 
data collections for elementary and secondary programs — each resulting 
in approximately 10,000 “person hours” for design, administration, 
collection, and reporting. From the vantage point of the department and its 
program offices, the information it receives has customarily been 
compromised because the schools, districts, and states reporting data 
employ their own definitions and, in some cases, report data that is 
inaccurate, incomplete, and not timely. Finally, in terms of program 
evaluation, much of the data that Education has traditionally requested 
has not necessarily focused on program performance. Yet the need for 
evaluative data has grown, particularly with passage of laws such as the 
No Child Left Behind Act of 2001, which requires states receiving 
assistance under the act to report on, among other things, the achievement 
of their students on academic assessments required under that law. 

To address these problems and better evaluate its programs. Education in 
2002 began an initiative to consolidate and improve the information it 
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requests from states on elementary and secondary education and to seek 
more consistency and quality in the data states supply. The Performance 
Based Data Management Initiative (PBDMl) is a large-scale effort within 
the department to combine more than a dozen separate data collections 
into a single collection system, and better focus the information Education 
requests from states by eliminating duplication, conflicting definitions, and 
information that is not useful for the evaluation of its programs. The 
PBDMl represents an important step forward for Education in its efforts to 
monitor the performance of the nation’s elementary and secondary 
schools. The initiative is also a large-scale undertaking for state education 
agencies, which are volunteering to help develop uniform data and test the 
new data collection system while they continue to meet their ongoing 
reporting requirements. The PBDMl was scheduled to begin phasing out 
the old data collections by September 2005, following final testing of the 
new system and training of department staff. 

In view of its importance and the inherent challenges, therefore, we have 
prepared a study under the authority of the Comptroller General to 
provide Congress with information about Education’s progress with the 
PBDMl. We have examined Education’s work to (1) define what 
performance-related data it will collect from states on behalf of the 
program offices, (2) assist states in their efforts to submit quality 
information, and (3) utilize performance-related data to provide enhanced 
analytic capacity within the program offices. 

To address our objectives we reviewed relevant documents, including 
Education’s business plans, information collected by Education on states’ 
capacity to supply data, various contracts for key pieces of the initiative. 
Education’s submissions to the Office of Management and Budget (0MB) 
justifying the various data collections, the department’s concept of 
operations, and other information related to the development of PBDMl. 
We also interviewed Education officials overseeing PBDMl, officials from 
most of the participating program offices, and key stakeholders in PBDMl, 
including a standards-setting organization, an advocacy group, and 
contractors, to obtain their perspectives on the progress of the initiative 
and to verify the information we reviewed. Finally, we surveyed 52 state 
data coordinators, including the District of Columbia and Puerto Rico, 
about their experiences with PBDMl, and we received 50 responses. We 
performed this work between April 2004 and September 2005 in 
accordance with generally accepted government auditing standards. See 
appendix 1 for additional information on our scope and methodology. 
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Results in Brief 



Through its PBDMI, Education officials have said that they have identified 
and defined much of the data to be collected under a unified data 
collection system. To determine what data will be collected, project 
officials have engaged in an ambitious effort with the program offices to 
identify data needed for program administration and oversight. They also 
developed performance-related data that would meet those needs, 
particularly for evaluating the effectiveness of federal programs. They 
further worked to develop common definitions and eliminate redundancy 
for data that would be collected through the system. The end result of this 
work was a body of performance-based data elements designed to better 
position the department to monitor the performance of its elementary and 
secondary education programs. However, officials responsible for the 
initiative told us that they were unable to resolve all data differences 
among Education’s program offices, given the traditional, diffused control 
of information collected throughout the department. PBDMI officials 
estimated that the majority of the work to define these data elements had 
been completed, although we found that they did not develop baseline 
data which would allow them to track the full extent of their progress. We 
were also told that these hard to resolve differences that remain would 
ultimately be settled at higher levels within the department, but the 
department has no formally agreed upon process for how or when such 
decisions would occur. 

PBDMI officials have conducted extensive outreach to the states to help 
them meet Education’s data request and in some cases upgrade their 
collection and submission systems; but after 2 years of testing, most states 
have not, for a variety of reasons, been able to provide Education with 
enough reliable data to proceed with the initiative. This outreach involved 
two rounds of site visits to all the participating states to confer about data 
elements developed with the programs and offer technical assistance, and 
$100,000 in grants to most states to help offset their costs. In addition. 
Education sponsored regional conferences and developed a call center to 
help states prepare and submit data. The department’s activities were 
focused largely on state-level agencies, but did involve some educational 
organizations. State data providers responding to our survey expressed 
general satisfaction with the department’s outreach, but about 75 percent 
nevertheless predicted that the burden of collecting and reporting data 
would increase or remain the same once PBDMI was completed. Many 
states also expressed doubts that PBDMI could enhance their analytic 
abilities. Only about 20 percent of states expected PBDMI to improve or 
greatly improve their analytic capacity. Despite the extensive outreach, 
most states were not able to produce enough data during test submissions 
in 2002-2003 and again in 2003-2004 for the department to validate its 
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quality and consider phasing out its standing collection systems. Thus, the 
department has decided to keep the latter collection phase open longer. 
According to PBDMl officials, some states lack the technical capacity to 
collect and report the requested data electronically and others need to 
modify their existing processes to meet the new specifications. Still others 
wanted clarification from the department for data definitions. State data 
providers also reported having competing demands for their time and 
attention, given other federal initiatives. 

Education officials have decided to proceed with PBDMFs implementation 
despite a shortage of data, other delays, and reservations among a few 
program offices; however, they do not have a specific plan for addressing 
these obstacles. Currently the department expects to complete its systems 
development efforts, which includes the full implementation of its data 
analysis and reporting system by the spring 2006 — 1 year later that its 
initial completion date — primarily due to the lack of state data and the 
failure of some of Education’s contractors to meet scheduled delivery 
dates. To the degree that it has been able to proceed, the department has 
begun developing a set of quality checks, although a few program offices 
expressed concern about their adequacy for maintaining the value of the 
data. Meanwhile, Education officials have said they are developing 
strategies to address these obstacles, including exempting states from 
certain reporting requirements, but they had no specific plan for providing 
further assistance to the states or for meeting state expectations for 
phasing out multiple data collections. 

We are making recommendations to Education to improve its planning 
and decision-making processes supporting PBDMl. In responding to a 
draft of this document. Education’s Assistant Secretary for the Office of 
Planning, Evaluation and Policy Development provided written comments 
on a draft of this report. In its comments. Education agreed with our 
findings and recommendations. Copies of the written comments are in 
appendix 11. 



Background 



The Department of Education annually administers data collections to 
gather information from states about elementary and secondary education 
programs receiving federal assistance. When it administers a data 
collection. Education, like most federal agencies, is required to follow the 
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provisions of the Paperwork Reduction Act (PRA)^ in order to maximize 
the utihty of information to the federal agency and minimize the level of 
burden incurred by the states and agencies from whom it solicits the 
information. Traditionally, the department’s program offices, which have 
responsibility for the administration and oversight of federal education 
programs, have developed and operated similar data collections 
independent of one another, in a continuous year-round process. In 
addition, much of the data requested from states has been focused on 
compliance and procedural matters, and overlooked performance and the 
impact of programs in the classroom. Moreover, the collection of this data 
has been complex and prone to error, given that it typically passes from 
about 94,000 public schools to more than 14,000 school districts and then 
to state education agencies before Education receives it. 

Collecting data can be both time-intensive and costly. Education estimated 
that in 2004, for example, that states spent approximately 45,000 hours and 
nearly $1.2 million responding to the department’s requests for certain 
elementary and secondary education data. (See fig. 1.) Data collections are 
costly for Education also. Over $5 million was spent in 2004 administering 
certain data collections that included allocating federal funds for both the 
staff to administer the collections and in many instances for contractors to 
analyze these data. 



^ The PRA was originally enacted in 1980 and most recently reauthorized and amended in 
1995 (Pub. L. No. 104-13, May 22, 1995). Generally, the law requires each agency’s chief 
information officer (CIO) to review program offices’ proposed collections to ensure that 
they meet PRA standards before submission to the Office of Management and Budget 
(0MB) for its approval. See also GAO, Paperwork Reduction Act: New Approach May be 
Needed to Reduce Government Burden on Public, GAO-05-424 (Washington, D.C.: May 20, 
2005). The scope of this report did not include a review of Education’s compliance with the 
PRA. 
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Figure 1 : Time and Money — Estimated Annual State Burden Hours and Costs for Select Elementary and Secondary Education 
Data Collections 



Gun Free Schools Act Report 

Title III, Biennial Evaluation Report Required of SEA'" 
Regarding Activities Under the NCLBA 

Title I, Part C, Education of Migratory Children: 
Migrant Child Count Report 

State Data Collection for the McKinneyA/ento 
Homeless Assistance Act 

Report of Children Receiving Special Education 
under Part B of the IDEA*’ 

Implementation of FAPE** 

Personnel Employed to Provide Special Education and 
Related Services for Children with Disabilities 

Children with Disabilities Exiting Special Education 
during the School Year 

Report of Children with Disabilties Removed, Suspended, 
or Expelled for More than 10 Days 

Part B of the IDEA Annual 
Performance Report 

Carl D. Perkins Vocational and 
Technical Education Act 

Common Core of Data Surveys 



12 10 8 6 4 2 0 

State burden hours (in thousands) 



Source: Department of Education. 

"State Education Agencies (SEA). 

"Individuals with Disabilities Education Act (IDEA). 

"Free and appropriate public education (FAPE). 

Note: Figure includes burden estimates for ongoing collections for which data were available. 
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Estimated state cost incurred responding to data collections 
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Initiated in 2002, the Education’s PBDMI has four goals: to improve the 
quality of the data Education collects about elementary and secondary 
education in terms of accuracy, consistency, and timeliness; to reduce the 
burden that states incur in reporting data to the department; to improve 
the focus of data analysis on program performance; and to improve 
Education’s data-sharing relationship with the states. While this initiative 
is not the department’s first attempt to overhaul the way it collects data, it 
nonetheless represents a fundamental change to its data management in 
that it is agencywide as opposed to program specific.^ As envisioned, the 
new collection would consolidate 16 separate collections heretofore 
conducted by seven program offices.^ Given the additional reporting effort 
that development and testing of the system would require of states. 
Education sought and received 0MB approval to collect data from the 
states through PBDMI.'‘ (See table 1 for a list of the separate collections 
the PBDMI is designed to supplant.) 



^ One earlier attempt, known as the Integrated Performance Benchmarking System, was a 
two-state demonstration project designed to consolidate department reporting 
requirements, but was terminated in 2000 without an assessment of its feasibility. 

^ Two collections that were formerly administered by the offices for Civil Rights and 
Special Education and Rehabilitative Services have already been subsumed into PBDMI. 

'' This approval will expire on September 30, 2005. Currently, Education is seeking approval 
for further PBDMI data collections beginning in 2006 through 2008. OMB’s approval of this 
extension of PBDMI data collection efforts was pending as of the end of August 2005. 
Additionally, any subsequent data collections would also be subject to the PRA process, 
including CIO review and 0MB approval. 
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Table 1 : Data Collections That Could Be Reduced or Eliminated as a Resuit of PBDMI 


Program office 


Data collections 


Office of Elementary and Secondary 
Education (OESE) 


1 . Consolidated State Performance Report 

2. State Data Collection for the McKinney-Vento Homeless Assistance Act 

3. Elementary and Secondary Education Act Title I, Part C Migrant Child Count 
Report 


Institute of Education Science (lES) 


4. Common Core of Data Surveys 


Office of English Language Acquisition 
(OELA) 


5. Title III Biennial Evaluation Report Required of State Education Agencies 
Regarding Activities under the No Child Left Behind Act of 2001 

6. Biennial Report Form for the Emergency Immigration Education Program 


Office of Special Education and 
Rehabilitative Services (OSERS) 


7. Report of Children with Disabilities Receiving Special Education under Part B of the 
Individuals with Disabilities Education Act 

8. Part B, Individuals with Disabilities Act Implementation of “Free and Appropriate 
Public Education” (FAPE) Requirements 

9. Personnel Employed to Provide Special Education and Related Services for 
Children with Disabilities 

10. Report of Children with Disabilities Exiting Special Education During the School 
Year 

1 1 . Report of Children with Disabilities Unilaterally Removed or Suspended/Expelled 
for More than 10 Days 

12. Part B of the Individuals with Disabilities Education Act Annual Performance Report 

13. Consolidated Data Collection on Students with Disabilities “ 


Office of Safe and Drug-free Schools 
(OSDFS) 


14. Gun-Free Schools Act Report 


Office for Civil Rights (OCR) 


15. Elementary and Secondary School Civil Rights Compliance ReporL" 


Office of Vocational and Adult Education 
(OVAE) 


16. Carl D. Perkins Vocational and Technical Act Annual Performance Report 




Source: GAO analysis of Department of Education documents. 

‘‘Collections that have been replaced by PBDMI. 




In addition to defining the information to be collected, the initiative 
involves the development of a Web-based, data exchange network that will 
provide states and others with the ability to submit school-based data into 
one unified system to be stored in a data repository. The network will 
comprise three separate, but interrelated systems — the first system, the 
submission system, developed in late 2004, is used to collect data from 
states, check data for quality, and store the data in the data repository. The 
second system, the survey tool, which was also developed in 2004, enables 
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Education to collect supplemental data from states and others that is also 
stored in the data repository. The third system, the data analysis and 
reporting system, which is not yet operational, will allow users (i.e., 
program office staff and the public) to among other things, query the data 
repository to analyze retrieved data and generate ad hoc reports. 
Education envisions that states and school districts would be able to use 
the data to assess their own program performance while also providing an 
opportunity for them to verify the quality of data submitted through the 
system. Figure 2 depicts the system design for the data network. 



Figure 2: Proposed Design for PBDMI’s Web-based Network 




Education’s data exchange network 



Data submission 
system 

receives data 
from states via the 
Internet 



Survey tool 

supplements data 
normally collected 
from states through 
submission system 



Schools 

The nation’s nearly 
94,000 schools 
are required to 
provide performance 
data to districts, 
states, or Education. 



Districts 

More than 14,000 
districts transmit 
their data to states 
or directly to 
Education. 



States 

States receive data 
from schoois and 
districts, aggregate 
it, and pass it along 
to Education through 
PBDMI’s Web-based 
submission system. 






Public 

Selected information collected from the 
states will be posted to the department’s 
Web site and made publicly accessible 



Data analysis 
and reporting system 

obtains data for analysis 
and reporting for use by 
Education, state and local 
agencies, and the public 



Central data 
repository 

stores data that has 
been checked for quality 



Source: Adapted from Department of Education materials; graphics by Digital Vision, PhotoDisc, and Art Explosion. 



Education had originally planned to have all components of the data 
exchange network fully operational in the spring of 2005 following the 
completion of key activities, such as (1) defining the data to be collected 
through in-depth consultations with department program offices and with 
state data providers, (2) populating the database with school-based data 
submitted by the states so that the quality of the stored data can be 
checked, and (3) training program staff on how to use the new network. 
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PBDMFs efforts to define what data were to be collected included forging 
agreements among Education’s individual program offices about which 
data would be essential to administration and oversight, particularly as 
performance indicators, and also developing common definitions for those 
elements that had been redundant. As a collaborative project, this involved 
developing consensus and receiving feedback from many parties-program 
offices, state policymakers and data providers, and organizations that 
develop data standards in the field of Education. Within the department, 
the office responsible for the day-to-day work of the project and for 
ensuring its success is the Strategic Accountability Service, which also has 
responsibility for developing and disseminating agencywide performance 
indicators. However, a number of other offices and boards within the 
department have been charged with providing oversight and guidance: a 
steering committee convened to share information on the development of 
the initiative consisting of the PBDMl managers and other senior officials 
within the participating program offices, the Chief Information Officer 
(CIO), a data information working group, and Education’s investment 
review board. The data information working group, which is headed by 
Education’s CIO, has responsibility for ensuring the consistency and 
quality of new data collections and for facilitating the integration and 
sharing of information between program offices. The department’s 
investment review board has overall responsibility for reviewing and 
approving and prioritizing department investments in technology, 
including the new network. As voluntary participants, stakeholders such 
as data coordinators from each of the 50 state education agencies, the 
District of Columbia, and Puerto Rico were provided with opportunities to 
give their input and feedback on the development of the initiative. The 
Education Information Advisory Committee established by Council for 
Chief State School Officers facilitates this exchange. Figure 3 depicts the 
various groups involved in the initiative. 
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Figure 3: Illustration of Key Actors Involved in Development of the PBDMI 




Source: GAO analysis. 



Once departmental data requirements were identified, Education planned 
a series of data collections to be followed by extensive testing of the 
quality of that data by the program offices. Specifically, Education planned 
to have states submit the newly defined data for the 2002-2003 and 2003- 
2004 school years. (States would voluntarily make these submissions to 
PBDMI while also maintaining their current multiple reporting obligations 
under Education’s program offices.) In conjunction with the program 
offices, PBDMI officials then anticipated validating and verifying the 
quality of the new data submitted using a number of checks and 
evaluations. Also at this time the development of the system that staff 
would use to analyze data and generate reports was to be fmahzed. Once 
these activities were completed, the program offices were to assess 
whether the new system would be an adequate substitute for their existing 
data collections. 



Education has projected that it would spend just over $30 million through 
2005 and initial estimates indicate that the data network will cost — 
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beginning in 2006 — just over $4 million annually to maintain. See figure 4 
for project time frames and projected costs through 2009. 



Figure 4: Fiscal Year 2002-2009 Funding for Education’s Data Management Initiative, including Key Activities Planned for 
Project Initiation through Implementation 



Dollars In millions 
14 



12 




Design and 
proposal 



Initial test of 
transmission 
and data 
collection from 
states 



Data collected 
and stored in 
system 



Data users 
given access to 
demonstration 
systems, data 
quality assessed 



2006-2009 (projected funding for annual operating costs) 



Education 
submits 
business case 
for PBDMi 

OMB provides 
$650,000 in 
funding for 
initiative 



' PBDMi staff 
conducts site 
visits to assess 
technical 
capabiiities of 
states and 
coliect data 

' PBDMi provides 
$50,000 grant to 
states submitting 
test data 



PBDMI staff 
conducts 
additional site 
visits to prepare 
states for data 
collection 

Second data 
collection to be 
undertaken in 
November 

Each state to be 
awarded another 
$50,000 for 
participation 

Sun/ey tool 
developed 



• Data analysis tools 
made available to 
Education's program 
offices 

• T raining for program 
offices and states on 
system use 

• Department to decide 
(by end of September) 
whether to proceed 
with initiative 



Projected annual operating costs of system 



Source: Department of Education. 



“Includes $2 million to develop a Web-based survey tool designed to collect supplemental data from 
schools, districts, and states. 
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Education Has Made 
Progress Defining 
Data to Be Collected 
under a Consolidated 
System, but Project 
Officials are Unable to 
Reconcile All 
Differences 



Education officials spearheading PBDMI told us they have made progress 
defining the data to be collected. To do this, project officials worked with 
the program offices to identify their existing data needs. They also worked 
with program offices to translate these needs into performance-related 
data, such as math and reading achievement scores for different groups of 
students. Officials told us they had eliminated data elements collected by 
the program offices that are more indicative of process than performance. 
PBDMI officials encouraged program offices to identify performance- 
related data by using requirements specified in laws such as the No Child 
Left Behind Act and using the goals in the department’s strategic plans. 

PBDMI officials also worked with the program offices to reach agreement 
on common definitions for the data elements selected and to eliminate 
redundancy. For example, some programs needed information on charter 
schools, and PBDMI officials coordinated efforts within the department to 
develop one standard definition for them. The end result of these efforts is 
a unified body of data elements that includes definitions for each of the 
data elements and identifies the program with primary stewardship over 
decisions about that element. According to one department official 
managing the initiative, this collection will improve the quality of the data 
by assuring more consistency in what states provide. 

Although PBDMI officials reported progress in identifying performance- 
related data and establishing common data definitions, project officials 
have not fully documented these achievements by establishing a baseline 
and thus cannot be certain of the full extent of the progress made toward 
achieving their goal to enhance the department’s focus on outcomes and 
accountability. For example, while PBDMI officials were able to provide a 
list of 161 data elements focused on performance they were unable to 
provide us with a comprehensive list of “process-oriented” elements that 
had been eliminated. Similarly, while PBDMI managers reported that the 
program offices had agreed to definitions for the bulk of the data 
elements — one official estimated that they reached agreement for about 90 
percent of the data — they could not provide us with a complete list of 
redundant elements that had been eliminated or those that remain because 
they had not tracked them. 

While PBDMI officials could not provide a full list of disputed data 
elements, they reported that some differences still remain among program 
offices. Although PBDMI officials encouraged the use of strategic plans 
and statutory requirements to justify the selection of performance-based 
data, they told us that program offices had final say over what data to 
collect. For example, one office uses similar although somewhat broader 



Page 13 



GAO-06-6 Education's Data Management Initiative 



criteria that allow it to collect “data that can be reliably obtained from 
states or that Education has a documented need for.”^ Additionally, 
according to initiative officials, some differences remain due to differences 
in legislative requirements for the particular programs, while others 
resulted from preferences of some offices to continue using the same 
definitions as in the past. 

Officials responsible for carrying out the PBDMl told us they were unable 
to reconcile all differences. Officials told us they were working with the 
program offices to reach agreements, but said the programs maintain 
primary control for defining their data needs and would make final 
decisions. Additionally, we were told by Education’s CIO, who is required 
to review all data collections and who has a primary role within the Data 
Information Working Group, that this office does not have a role in 
resolving data disputes between program offices in order to ensure 
uniformity. However, an official also said that any differences that could 
not be resolved between the program offices would ultimately be 
arbitrated at the assistant secretary level within Education. 



PBDMl Officials Have 
Worked Extensively 
with States on Data 
Preparation and 
Submissions, but 
Most States Cannot 
Produce the 
Requested Data 



PBDMl officials have conducted extensive outreach to the states to help 
unify their data definitions and upgrade their collection and submission 
systems. State data providers responding to our survey expressed general 
satisfaction with the department’s outreach. However, the majority 
thought that the burden of data collection and reporting would either 
increase or remain the same with implementation of the PBDMl. In 
addition, less than half expected the initiative to improve their ability to 
conduct their own in-state analyses only somewhat. Despite the extensive 
outreach, the states were not able to produce enough data during test 
submissions in 2002-2003 and again in 2003-2004 for the department to 
validate its quality and consider phasing out its standing collection 
systems. 



^ Office of Elementary and Secondary Education, “Criteria for Inclusion of Data Elements 
in the Consolidated State Report,” 2005. 
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Education Has Conducted 
Extensive Outreach to 
States to Improve Data 
Quality 



In order to ensure that states could meet Education’s requests for quality 
data required as part of PBDMI, officials conducted extensive outreach to 
state agencies, their data providers, and to data standards organizations. 
After Education developed its body of data elements, it consulted in 2002 
with a task force consisting of a small number of state data providers to 
advise the department on the availability of the data it intended to collect. 
The department then conducted site visits beginning in April 2003 to 50 
states, the District of Columbia, and Puerto Rico to obtain feedback on the 
ability of states to provide needed data and to prepare for testing the 
states’ ability to submit data. Education officials said they also made 
$50,000 grants to all 52 states to offset costs of overhauling information 
systems or obtaining additional staff At the culmination of these visits. 
Education originally planned for states to transmit 2002-2003 school year 
data that could be tested for quality. 

However, Education scaled back the scope of this first data collection 
after recognizing that states would not, as yet, be able to offer certain 
types of data, such as data needed to meet requirements of the NCLBA. 
Consequently, Education delayed its plans to assess the quality of the data 
states submitted and focused instead on the ability of states to 
electronically transmit as much PBDMI data as they could to the 
department. Also, Education decided to remove from PBDMFs prospective 
collection some data elements that states reported were not available at 
that time. Under this transmission pilot test, 50 states, including the 
District of Columbia and Puerto Rico, were able to submit some data to 
Education demonstrating that PBDMI was technically feasible. 

After establishing this technical feasibility. Education began preparing in 
2004 for its data collection of 2003-2004 school year by providing 
additional outreach to the states. Project officials conducted a second 
round of site visits beginning in April and provided further guidance to 
help states align their data definitions with PBDMI standards. By aligning 
definitions with PBDMI, Education attempted to minimize possible 
confusion about what data to submit and when, further assisting the 
department’s efforts to improve data quality. Department officials have 
said that establishing a unified body of data elements across the 
department and states — so that all involved parties use the same 
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“language” when analyzing and sharing data® — is a priority. Education 
officials attribute the lack of quality in the data it currently collects from 
states and others to a variety of reasons, such as the lack of common data 
definitions that developed over time in response to the specific 
information needs of the program offices and data requirements arising at 
the state level. 

Officials with the initiative also conducted a limited number of quality 
assessments of state information systems to identify better ways of 
collecting and reporting data to the department. To serve states on a 
broader scale, Education conducted regional meetings, providing them 
with updates and feedback on the progress of the initiative. Officials also 
established a call center to answer states’ questions about the data to be 
submitted. Most states also received another $50,000 in grants for their 
continued participation in the initiative.^ Education began collecting 2003- 
2004 school year data in November 2004. 

To increase the likelihood that its definitions would be adopted by states 
and other data providers, PBDMl officials also collaborated with advocacy 
groups that establish data and influence the development of technical 
standards. For example, PBDMl officials contracted with the Council of 
Chief State School Officers to coordinate PBDMl conferences, help states 
prepare and submit data, and provide feedback as PBDMl developed data 
definitions. Education also collaborated with the Schools Interoperability 
Framework, a group that develops data-sharing standards and software 
primarily designed for schools and districts. By working with the Schools 
Framework, Education officials said they could improve data quality by 
increasing the likelihood that departmental definitions and other 
standards would be incorporated into software used by schools and 
districts. This interaction with the Schools Framework is Education’s 
primary attempt to deal with the long-standing problem of poor data 



® Some of the work to establish a consistent standard has been ongoing throughout the 
department prior to the development and implementation of PBDMl such as the work 
undertaken by Education’s National Center on Education Statistics to establish a 
departmentwide data dictionary. PBDMl has a role in contributing to some of these other 
efforts. 

’’ According to department officials, all 50 states, the District of Columbia, and Puerto Rico 
accepted grants in 2003; and in 2004, 46 states acquired another $50,000. The grants were 
awarded solely on the basis of state participation in PBDMl, and states were allowed wide 
latitude in their usage. 
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provided by schools and districts.*^ (See table 2 for a list of some of 
Education’s outreach activities.) 


Table 2: Education’s Outreach Activities to improve Data Quaiity under the PBDMI 


Activity 


Description 


Purpose 


Site visits (1st round) 


Met with states in summer 2003 to discuss data 
definitions and availability for test data coiiection in 
November 2003. 


To introduce standards, encourage 
consistency, and assess data avaiiability 
and technicai capacity of state 


Site visits (2nd round) 


Met with states in spring 2004 to confirm data 
definitions and availability for initiai data coiiection in 
November 2004. 


information systems. 


$50,000 PBDMI Participation 
Grants 


Officials report awarding 52 state education agencies 
with funds in 2003-2004; and grants to 46 states 
received funds in 2004-2005. 


To obtain state buy-in and offset costs. 


State taskforce 


This advisory group, made up of a smali number of 
states, has provided input concerning available data 
that can be submitted by states and how coiiections 
couid yield better quality data. 


To solicit initial state input on data 
issues such as availabiiity and capacity. 


Technicai assistance 


Cail center avaiiabie to ail states, meetings, 
conferences, and data quality assessments provided 
to 1 0 voiunteer states. 


To provide states with answers to 
questions, updates on the status of the 
initiative, and information to heip 
improve data systems. 


Outreach to software vendors 


Education coordinates with key standard setting 
organizations such as the Schools Framework. 


To work coilaborativeiy to develop 
educational data standards. 


Source: Department of Education. 



States were generally satisfied with Education’s outreach activities. (See 
table 3.) Most state data providers — 72 percent — rated Education’s site 
visits effective in improving the partnership with the states. One state data 
provider characterized his exchanges with the department as open and 
non-defensive, and further reported that the department had been 
responsive. More than half rated as effective or very effective Education’s 
technical assistance (57 percent) and regional meetings (52 percent). 



Most States Expressed 
Satisfaction with 
Education’s Outreach, but 
Had Mixed Views on 
PBDMI’s Potential Benefits 



* Past reports issued by GAO (published jointly with other education agencies) and 
Education’s Inspector General (IG) document that inadequate data quality practices by 
schools and districts have adversely affected the states’ ability to produce quality data. In 
2002, GAO and Education’s IG reported that states had problems entering accurate data 
and lacked sufficient supervisory review procedures to check data received from schools 
and districts. GAO et al., A Joint Audit Report on the Status of State Student Assessment 
Systems and the Quality of Title I Accountability Data, SAO-02-064, (Austin, Tex.: 2002). 
OIG, Department of Education, Improving Title I Data Integrity for Schools Identified for 
Improvement, ED-OIG/A03-B0025 (Philadelphia, Pa.) March, 2002. 
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While most states thought Education’s activities to improve its partnership 
with states were effective, some suggested areas for improvements. For 
example, 72 percent thought the site visits provided only some or little 
information on successes achieved in other states. 



Table 3: State Survey Responses on Education’s Outreach Activities through PBDMI 


How effective has Education been in its goai to improve coordination with states? 


Very effective/effective 


Ineffective/very ineffective 


Site visits 


72% 


8% 


Technical assistance 


57 


4 


Regional meetings 


52 


6 


To what extent did the 2004 site visits provide the following benefits? 


Very great/great extent 


Little to no extent/some extent 


Opportunity to share information 


50% 


24% 


Opportunity to provide feedback 


44 


20 


Obtained information about successes in other states 


6 


72 


Obtained a better understanding of the benefits to states 


30 


34 



Source: GAO, Survey of States on the Department of Education’s Performance Based Data Management Initiative (PBDMI). 



In their survey responses half of the states expressed the view that 
reducing their reporting burden was the most important PBDMI goal; 
however, fewer than a third of the states said they believe the initiative 
will do so. (See table 4.) Some states emphasized their burden had 
increased in the short term as they continued dual reporting in order to 
meet the still ongoing data collection requirements of the program offices. 
Three states reported to us their cost estimates of systems development 
projects needed to support PBDMI, which ranged from approximately 
$120,000 to as much as $5 million. Moreover, about 75 percent of the states 
reported that they thought the burden to collect data would remain the 
same or increase once PBDMI was implemented. Some state respondents 
expressed the opinion that until there is a firm commitment by Education 
to halt multiple data collections their reporting burden would not likely 
lessen. “We are asked from the federal government for more and more 
information. ... [which] opens the flood gate for more and more reporting,” 
noted one official, adding that it is currently “hard to see the benefit at this 
time.” 
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Table 4; Percentage of States Reporting Which Goais Were Most Important, Attainable, and Difficult to Achieve 



PBDMi goais 


Most 

important goal 


Most 

attainable goal 


Most difficult 
goal to achieve 


Reducing collecting and reporting burden 


50% 


30% 


36% 


Improving data quaiity 


26 


16 


30 


Improving the partnership with states based on 
common data standards 


12 


24 


14 


Focusing on outcomes 


0 


16 


8 


Not sure 


12 


14 


12 



Source: GAO, Survey of States on the Department of Education’s Performance Based Data Management Initiative (PBDMI). 



Some states also had reservations about the benefits of PBDMI for 
evaluation. One respondent cautioned, for example, that support within 
his state had weakened because of the lack of perceived benefits. Only 
about 20 percent of states expected PBDMI to improve or greatly improve 
their analytic capacity — that is the ability to meet their own state reporting 
requirements, analyze program effectiveness, analyze student outcomes, 
and to compare outcomes within states. Their reasons varied. For 
example, five states reported that they would continue to use their own 
systems. A few elaborated that their own information systems allow more 
detailed analyses of state performance than the information to be 
collected through PBDMI. Additionally, an almost equal number of states 
saw PBDMI as an effective tool to inform stakeholders as not. Table 5 lists 
the extent to which state data providers expect PBDMI to enhance their 
analytical capacity in a variety of areas. 



Table 5: Percentage of States Reporting the Extent to Which PBDMi Would Improve Their Analyticai Capacity 





Very great/great extent 


Little to no/some extent 


Inform stakeholders 


35% 


33% 


Meet state reporting requirements 


22 


59 


Analyze student outcomes 


22 


53 


Make budgetary decisions 


16 


61 


Analyze program effectiveness 


20 


45 


Compare outcomes within states 


20 


51 



Source: GAO, Survey of States on the Department of Education’s Performance Based Data Management Initiative (PBDMI). 
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Many States Are Not 
Prepared to Meet 
Education’s Data Requests 



As of June 3, 2005, only 9 states had submitted more than half of the 
requested 2003-2004 school year data, while 29 states had submitted less 
than 20 percent (see fig. 5). Although PBDMl officials said they will wait 
until August 2005 for states to submit the 2003-2004 data, they also 
acknowledged that many states would not be able to provide significant 
portions. The lack of state data is particularly acute in some programmatic 
areas. For example, many states have been unable to provide data on 
homeless and migrant students or students with limited English 
proficiency. States told Education officials early in the process that 
changes to state data collection processes, systems, and definitions would 
be needed to provide these types of information. 
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Figure 5: School Year 2003-2004 Performance Data Submitted by States to PBDMI as of June 3, 2005 




Hawaii 



Puerto Rico 



States that submitted 51 to 83 percent of requested performance dafa 
Stafes fhat submitted 21 to 50 percent of requested performance dafa 
Stafes fhaf submitted 3 to 20 percent of requesfed performance dafa 



Source: Department of Education. 



We found that there were various reasons why states could not provide 
data. Some states reported that they wanted better documentation from 
the department in areas such as clarifying established data definitions and 
file format specifications needed to transmit data. States needed to make 
major modifications to their existing data collection and reporting 
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processes in order to provide new information required by PBDMI. States 
also reported that they would not provide certain data elements that were 
inapplicable, hard to collect, or available elsewhere. Some also reported 
that there was still some confusion over multiple or unclear definitions. 
Department officials said that many states had initially overestimated their 
capabilities and that the data states said would be available d iff ered 
greatly from what they have produced thus far. States have also noted 
competing demands for their time and resources stemming from NCLBA. 
Some states reported they lacked resources, such as staff and money, to 
implement changes specific to the initiative. Specifically, 56 percent of the 
state survey respondents said that all or a portion of the $50,000 in grants 
they received from Education were used to contract for additional 
personnel, a quarter of the states said that these funds were used to 
improve their information systems. Some states noted, however, that these 
funds were insufficient to make changes necessary for their participation 
in PBDMI. 

Recognizing that obtaining state data has been problematic. Education has 
recently developed a preliminary strategy for working more closely with 
states to ensure that it obtains 100 percent of data from all the states. 

While not finalized. Education is currently considering actions such as 
issuing regulations requiring states to submit PBDMI data and allowing 
those states that provide acceptable amounts of “high quality” data under 
PBDMI to be exempt from existing data collections. For example, states 
that submit data to PBDMI that are also currently collected through the 
Consolidated State Report — one of many data collections required under 
the NCLBA — would not have to submit the same data under this data 
collection. Officials have also tentatively proposed collecting data of lesser 
quality that are readily available and obtaining data through other systems 
to supplement what has been provided thus far. It is not clear the extent to 
which this proposal would undermine efforts to improve data quality and 
maintain program office buy-in. Another option under consideration at 
Education is to target departmental resources, such as $25 million in 
grants for system improvements from the Institute of Education Sciences, 
at states that actively participate in PBDMI. 
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Education Is 
Proceeding with 
Implementation 
despite the Data 
Shortage and without 
a Detailed Plan of 
Action 



Education is proceeding with efforts toward full implementation of 
PBDMl — using the data for analysis and reporting — despite the limited 
amount of data collected. To do so, program offices decide whether the 
quality of the data (in terms of accuracy, consistency, timeliness, and 
utility) collected through PBDMl meets their needs. Once program offices 
validate the quality of the data. Education would begin to phase out 
existing data collections. Additionally staff will be trained on how to 
access and use the data collected to date. Originally Education expected 
to complete all of these activities by the spring of 2004. To the degree that 
it has been able to proceed, the department has developed a set of quality 
checks designed to ensure the accuracy and completeness of the data 
states submit. 



Nevertheless, two program offices, which as members of the seven 
principle offices included in the initiative and have a role in determining 
whether the data are accurate and complete for their purposes, expressed 
concern that PBDMFs procedures to ensure data quality may not be 
adequate. An official in the Office of Special Education and Rehabilitative 
Services (OSERS), which has collected almost 30 years of longitudinal 
data about the effectiveness of the nation’s special education programs, 
told us that PBDMl had been provided with information about the nearly 
200 data quality checks used in special education collections, but was not 
sure that PBDMl adopted them all. PBDMl officials said they adopted 
those that were universally relevant. Further, this official expressed 
concern that PBDMl would not meet its special needs. Specifically, unlike 
other program offices, OSERS programs bases student assessment on age 
as opposed to grade level attained. Additionally, this official was 
concerned about the timeliness of the data collected through PBDMl 
because that office generated a number of congressionally mandated 
reports at specific times of the year. Consequently, this office plans to 
compare the quality of its own data with the data collected through 
PBDMl. Officials in the Office for Civil Rights also expressed similar 
reservations with PBDMFs administration of its large elementary and 
secondary survey of schools and districts used to assess compliance with 
civil rights laws and identify trends. Historically, district superintendents 
have responded to this survey in large enough numbers allowing 
Education to generalize on any findings with a high degree of confidence.” 
However, when PBDMl administered the survey, fewer superintendents 



” Officials have told us that historically 97 percent or more of randomly selected school 
districts have responded to this survey. 
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responded and, according to the Office for Civil Rights, PBDMI did not 
have a readily available plan that adequately outlined steps needed to raise 
the response rate. As of June 10, 2005, the response rate for this survey 
was lower than previous surveys. 

Final implementation has also been hampered by delays in training and 
delivery of the analysis and reporting system. Both are more than a year 
behind schedule. An official responsible for overseeing the training efforts 
told us they could not focus on the delays because considerable time was 
spent addressing state problems submitting data through PBDMI. The data 
analysis and reporting system is more than a year behind scheduled due to 
the lack of data and the failure of Education’s contractor to meet its 
scheduled delivery of the system. Education officials now expect to fully 
implement the system by March 31, 2006. In lieu of developing its data 
analysis and reporting system and training, PBDMI has offered 
presentations of these tools as a preview for staff to see the new system’s 
capabilities and to keep them apprised of the initiative’s progress. 

Despite the many obstacles confronting the PBDMI, Education officials 
said they expect to proceed with implementation of the initiative, albeit 
with some activities postponed. In August, project officials developed a 
preliminary strategy designed to address the problem of collecting data 
from the states, such as providing exemptions from certain reporting 
requirements for some states. However, this strategy has not been 
finalized, and Education has not developed a specific plan of action for 
how they might (1) help states that are deficient, (2) deal with state 
expectations for phasing out the multiple data collections, or (3) meet the 
expectations of their own program offices. 



Conclusions 



The PBDMI represents an important step forward for the Department of 
Education in its efforts to monitor the performance of the nation’s 
elementary and secondary schools. By developing the ability to collect 
data that are more accurate, timely, consistent, and focused on key 
national performance indicators. Education will be much better informed 
to make its many policy and programmatic decisions. The initiative, by 
asking for a clearly defined set of information that is to be submitted only 
one time, has the potential to substantially reduce state reporting burden 
for elementary and secondary programs as well as to help states to 
develop better data systems. However, PBDMI is an ambitious and risky 
undertaking that requires the continued cooperation of a number of 
internal and external stakeholders. 
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In order for PBDMI to be successful, the department must rely on states to 
provide new information at a time when they are busy implementing large 
new federal initiatives, such as the No Child Left Behind Act. While some 
states have been able to provide significant amounts of data, others 
continue to lag far behind. In order for PBDMI to be successful, it is 
important for all states to submit timely, reliable, accurate, and consistent 
data. Consequently, it is important for the department to have a clear plan 
for addressing states with problems providing data and to continue to 
provide a proper combination of support and incentives for states to 
participate. By having worked closely with the states on their collection 
systems, PBDMI officials have the information they would need to develop 
a plan of action to help move them forward. 

Because PBDMI represents a significant change in the way the Department 
of Education conducts business, it can only be accompanied effectively 
and efficiently by a change in management practices. However, program 
offices still retain much discretion over what data they will collect, how 
they will define it, and whether or not PBDMFs data will meet their needs. 
While it is the initiative’s responsibility to make sure it collects data that 
meets the program offices’ requirements, PBDMI is also responsible for 
developing a data collection system focused on program performance and 
quality data. To the extent that programmatic differences, such as those 
over data definitions, inhibit PBDMFs goals there should be a clear 
process for reconciling those differences. If PBDMI truly represents a new 
way of doing business. Education should be able to ensure that its 
organizational units go along. It is difficult to see PBDMI achieving its full 
potential without a clear process for furthering the initiative’s goals. 

Fundamental to any large, complex effort’s success is a well thought out 
plan that tracks its progress against a set of clearly defined and 
measurable goals. PBDMI has not put in place such a planning and 
tracking system. State governments and Education’s program offices have 
devoted much time, effort, and money participating in PBDMI with the 
idea that they would see benefits as a result. A lack of demonstrated 
progress and benefits potentially erodes state support, undermining the 
viability of this important initiative. Some states are already beginning to 
lose sight of the potential benefits of PBDMI. As the department goes past 
its original completion deadline, it is important for it to lay out a clear plan 
for how it will now proceed. 
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RGCOrn.rnGndcltionS for address the issues we have identified with regard to planning, decision- 
making, and improving data quality, we recommend that the Secretary of 
ExGCutivG Action Education develop 

• a strategy to help states improve their ability to provide quality data 
given the challenges that many states face in providing data; 

• a clear process for reconciling differences between the program offices 
and the PBDMl oversight office to ensure that decisions critical to the 
success of PBDMl are made; and 

• a clear plan for completing final aspects of PBDMl, including specific 
time frames and indicators of progress toward the initiative’s goals. 

We received written comments on a draft of this report from the 
Department of Education. Education agreed with our findings and 
recommendations and stated that it has devoted additional resources to 
the initiative and plan to issue a detailed project plan that outlines the 
steps needed to complete the initiative. These comments are reprinted in 
appendix 11. 

Education also provided technical corrections and comments that we 
incorporated where appropriate. 



We are sending copies of this report to the Secretary of Education, the 
Office of Strategic Accountability Services, the Director of the Office of 
Management and Budget, and appropriate congressional committees. 
Copies will also be made available to other interested parties upon 
request. Additional copies can be obtained at no cost from our Web site at 
www.gao.gov. 

If you or your staff should have any questions, please call me at 415-904- 
2272 or bellisd@gao.gov. Contact points for our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this report. 



AgGncy ComrriGnts 
and Our Evaluation 
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GAO staff who made major contributions to this report are listed in 
appendix III. 




David Beilis 

Director, Education, Workforce 
and Income Security Issues 
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Appendix I: Scope and Methodology 



The objective of our review of the Performance Based Data Management 
Initiative (PBDMI) was to assess the progress Education has made in its 
implementation of the initiative, particularly with regard to (1) defining 
what performance-related data it will collect from states on behalf of the 
program offices, (2) assisting states in their efforts to submit quality 
information, and (3) utilizing performance-related data to provide 
enhanced analytic capacity within the program offices. We conducted our 
review between April 2004 and September 2005 in accordance with 
generally accepted government auditing standards. 



Overall Approach To assess the department’s progress in each of these areas, we reviewed 

documents relating to the implementation of the initiative, relevant laws, 
and information provided by the office responsible for PBDMI — the 
Strategic Accountability Service (SAS) and others. We interviewed key 
staff responsible for the initiative as well as officials in each of the offices 
that are participating in PBDMI. We also interviewed senior-level 
Education officials to determine their role in the implementation of 
PBDMI. To gain insight into state perspectives on the initiative, we 
administered a Web-based survey to state officials responsible for 
providing these data to Education. We received responses from 50 states 
including Puerto Rico. We also interviewed a variety of external 
stakeholders, a data standards organization, and three contractors 
involved in the initiative, including an official from the Council of Chief 
State School Officers. We also reviewed previously issued reports by 
Education’s Office of the Inspector General (IG) as well as GAO reports 
and testimonies. 

In addition to interviewing departmental officials, we also reviewed 
documentation on the initiative to gain a better understanding of what 
actions Education was undertaking to implement the goals of the 
initiative, including its data quality contract, data dictionary, its business 
plans as well as justification reports to the Office of Management and 
Budget (0MB) required under the Paperwork Reduction Act to collect 
data. We also reviewed summary information about state performance 
data that was obtained as a result of site visits to states conducted in 2004 
in order to analyze what data was obtained from states as a result of their 
efforts. 

Education provided information on states’ submission of requested data 
elements to PBDMI as of June 3, 2005. States were expected to provide 
data for 64 data elements ranging from dropout rates, student performance 
on reading, science, and writing assessments, teacher certification, and 



Page 28 



GAO-06-6 Education's Data Management Initiative 






Appendix I: Scope and Methodology 



many others. For each of these elements, Education determined whether 
each state had submitted the information, had not submitted the data, or 
did not collect the information. We incorporated into our report 
Education’s calculated percentages of elements submitted for each state. 
We determined that these data were sufficient for the purposes of this 
engagement. 

In order to document the burden hours associated with certain elementary 
and secondary data collections, we accessed 14 data collection 
justifications authored by each of the department’s program offices and 
submitted to the chief information officer. These reports had received 
0MB approval or were seeking approval to collect data from states and 
others. We talked with an official responsible for maintaining these 
documents at the department’s Web site to verify that these were the most 
recent data available for analysis. From each document we obtained the 
estimated state burden hours and costs and federal administrative costs 
associated with each data collection. Each estimate was based on a 
formula that we adjusted to reflect these costs for the 52 states 
participating in the initiative. In some instances where an average was 
used, we assumed that the 52 states were similar in characteristics to the 
overall population of states included in Education’s estimates. However, 
we did not find it feasible to prorate the formulas for the federal 
administrative costs (based on 52 states) for each of the collections. A 
statistician verified each of the calculated estimates for accuracy. 

We also surveyed all 52 state data coordinators using a Web-based survey 
instrument in order to obtain their perspectives on various aspects of the 
initiative. Our survey instrument was developed based on information 
obtained during interviews with state data coordinators in Pennsylvania, 
Virginia, Washington, and Oregon. Additionally, other internal 
stakeholders specializing in technology and education were asked to 
review and comment on our draft survey instrument. The survey was pre- 
tested with Wyoming, North Carolina, and Illinois to determine if the 
questions were clear and unbiased and whether the terms were accurate 
and precise. We included these three states in our pretests because they 
varied in size and technical capacity for data transmission as determined 
by an earlier Education survey. Based on their comments, we refined the 
questionnaire as appropriate. 

Our final survey instrument asked a combination of questions that allowed 
for closed-ended as well as open-ended responses and included questions 
about state perspectives on PBDMFs ability to achieve its goals. The 
survey was conducted using self-administered electronic questionnaire 
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posted on the Internet. We sent e-mail notifications about the upcoming 
survey to all 52 state data coordinators (50 states, the District of Columbia, 
and Puerto Rico) on November 15, 2004, and activated the survey shortly 
thereafter. Each potential respondent was provided a unique password 
and username by e-mail to limit participation to members of the target 
population. To encourage respondents to complete the questionnaire, we 
sent an e-mail message to prompt each non-respondent approximately 2 
weeks after the survey was activated and followed up by e-mail or phone 
with each non-respondent several times thereafter. We closed the survey 
on January 21, 2005, after the 50th respondent had replied. 

Because this was not a sample survey, there are no sampling errors. 
However, the practical difficulties of conducting any survey may introduce 
errors, commonly referred to as non-sampling errors. For example, 
difficulties in how a particular question is interpreted, in the sources of 
information that are available to respondents, or in how the data are 
entered into a database or were analyzed can introduce unwanted 
variability into the survey results. We took steps in the development of the 
survey instrument, the data collection, and the data analysis to minimize 
these non-sampling errors. For example, a survey specialist designed the 
survey instrument in collaboration with GAO staff with subject matter 
expertise. Then, as stated earlier, it was pre-tested to ensure that the 
questions were clear, unbiased, and accurate. When the data were 
analyzed, a second, independent analyst checked all computer programs. 
Because this was a Web-based survey, respondents entered their answers 
directly into the electronic questionnaire, eliminating the need to have the 
data keyed into a database, thus removing an additional source of error. ^ 



^Source: GAO Intranet, ARM Guidance, “Evaluating and Reporting of Non-sampling Errors 
in Surveys.” 
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UNITED STATES DEPARTMENT OF EDUCATION 



OFFICE OF PIANNINC. EVALUATION AND PCSJCY DEVELOPMENT 



October 1 7, 2005 



ASSlSTANr sfx:retary 



David D. Beilis 

Director, Education, Workforce, and Income Security Issues 
United States Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 



Dear Mr. Beilis: 

Thank you for providing the Department of Education with a draft copy of the U.S. 
Government Accountability Office’s report entitled, “Education’s Data Management 
Initiative: Significant Progress Made but Better Planning Needed to Accomplish Project 
Goals” (GAO-06-6). We agree with your recommendations and value your observations 
on this continuing initiative. 

The Department’s performance-based data management initiative is essential to the 
Department’s efforts to improve the use of data to inform policy making and program 
management, to increase the focus on student achievement outcomes rather than on 
process, to reduce the burden on States of repotting data to the Department, and to 
improve the accuracy, timeliness, and utility of data collected. As you note in the report, 
the initiative is an ambitious undertaking, and we recognize the important challenges to 
achieving our goals. 

You recommend that the Department develop 1) a strategy to help States improve their 
capability to provide quality data, 2) a process within the Department to resolve critical, 
outstanding issues, and 3) a clear plan for completing final aspects of the initiative. We 
have also identified these as priority focus areas and began to address them shortly after I 
joined the Department this summer. Since that time, we have committed additional 
leadership and financial resources to the initiative, among other actions, and have begun 
developing the detailed project plan needed to successfully complete the initiative. We 
will include a full description of plarmed actions in our corrective action plan. 



Sincerely, 

/ 






Tom Luce 



400 MARYLAND AVE.. SW.mSHQNCTCW. DC2<«03'3200 
www.«i.J5*w 
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examines the use of public funds; evaluates federal programs and policies; 
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